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ABSTRACT o 
Disseip.ination- and Utilization of Assessment 
• Results in Oregon 

Inproved dissenination'' and utilization of results from statev/ide assessment,, 
of student performance is critically iieeded. Prior research shows that .less , 
than one third of all states can provide evidence of using such data to nake 
significant classes of decisions most frequently cited as a justification for 

o" 

the initiation of such programs* Yet the annual budgets for assessment programs 
continue to escalate. The evidence suggests that while statewide assessment programs 
are frequently initiated as part of accountability movements, they are seldom 
accountable themselves. 

In an attempt to deal with this problem, the Oregon assessment program has 
initiated a variety of innovati.ve steps designed to insure effective dissemination 
and maxinun utilization of annual testing results. Some of these efforts incldde: 
extensive interviewing and research to estaBlish.a <-lear purpose, foi the progr^an; 
identification of major policy questions to be impacted by assessment results; 
prior research to determine the most f^ffective dissemination modes for selected 

decision-ziaking audiences; widespread use of professional and lay advisory committees 

* 

in the design, conducting, interpretation and reporting of data; simultaneous re- 
lease of results, interpretations and recommendations for actions to specific audiences 
initiation of a state management system to systematically utilize assessment 
findings in specific decisions; continuing evaluation of all dissemination and 
utilization strategies; and follow-up contacts^ with key decision-makers to assure 

» 

utilization. • . ' 
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For the past two years, I have been collecting data for a comparative 
survey of statewide assessment programs throu2h6.ut the country. One significant 
finding from this study was that x^hile 69 percent of the states indicate that 
they use their statewide assessment results to establish priorities for Title III 
of the Elementary and Secondary Education Act, less than one third of the states o 
could provide evidence pf using this data to make other ty{$es of decisions which are 
frequently cited as a justification for the initiation of statewide assesment 
programs.^ . . . , ^ 

Compare these findings with the results of a recent survey by the U.S. Office 
of Education, where it was estimated that State Departments of Education are spending 
a minimum of $5 million annually to produce student assessment data required for 
decision-making under Titl§ III, ESEA.^ 

The development of statewide^ assessment programs are usually tied, to the move- 
laent for accountability within public education. But really, just how ^ccountable 



are statewide assessment programs themselves? The two topics I will speak to, 
dissemination and utilization, abpear to me to be the key to answering this question. 

Successful utilization of state assessment data is primarily. dependent on 
clearcut agreements about the purposes of the program and the decisions which will 
be made as a result of assessment findings. 

Oregon's first steps to define the purpose of its statewide assessment program 
were taken in the fall of 1972 when the University of Oregon, under contract to the 



1 Hall, Mary, "Statewide Assessment of Student Performance: A Comparative Study,'* 
presented at Symposium on Advances in the Methodology of Statewide Assessment, 
iVmerican Educational Research Association, 1975 Annual Meeting, Washington, D.C. 
p. 1:2. 

2 Hershkowitz, Martin, Statewide- Educational Needs Assessment; Results from Selected 
Model States , Hershkowitz Associates, Sjlver Springs, Maryland, 1974, p. 7. 



Department, conducted a series of 45 interviews to determine the .types^'of educational 
decisions which might be made using student assessment results. The* population 
for^ these interviews was chocen on a very Scientific basis ~ "Power." The individuals 
■ were those vho wexe felt to be "movers and shakers" within the state and national 
legislature, state governing boards, school superintendent organizations, universities. 



/ 
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teacher organizations, research and development organizations, local school boards 

and the Departmen^^tself • From these interviews, the University recommended that 

major purposes be established for the program, of which five pur^^es related to 

decision-making by the State Board of Education and five to decision-making by the 

Department and the Legislatur e»3 ^he keying of program purposes to specific decision-. 

« 

making audiences is a point which I will want to address later^ These recommendations 
were used as the basis for presenting the initial program request to the 1973 Oregon 
Legislature. They were al^o used as the basis for planning the. first pilot assessment 
for like-'most stateS'^ Oregon has planned and implemented its assessment concurrently. 

In 1974, the ass^essment staff further refined these 10 statements of general 

i 

purpose into a set of sixteen "policy questions" whicK the program might potentially 
attempt to answer. A questionnaire on these "policy questions" was sent to a variety 
of decision-making audiences throughout the state - this time including key statewide 
advisory committees charge^ with making program recommendations to the Board of 
Education for specific audiences (i.e., deaf, blind, migrant and so forth). .Results 
were tabulated both by totkl rank (i.e., those policy questions thought to be most 



3 Mattson, Robert H., A Conceptual Model for Statewide Assessment , report to the State 
Department of Education from the University of Oregon's College of Education 
FebrOary, "1973, pp. 29-30.. * ■ 
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iiaportant to everyone) and within response groups. This gave us even further 
data on the decisj.on-making needs of special bodies.^ 

(. 

^ These two sources of interview and questionnaire data, coupled with discussions 
with the Statewide /sseasment Afjvisory Committee, the State Board of Education and 
technical 'advisory committees, were further refined into four statements of purpose 
for the program,-^ which are now included in the long-range plan currently pending 
before the Oregon Legislature^ 

While establishing agreed-upon purposes for statewide assessment was an impor- 

* ' ' ' 

tant first step, an equally important step to Oregon was the creation of specific 

. decisiqn-making processes to insure that the assessment data was actually used. 

Beginning in 1972, the Oregon Board of Education had already made a commitment 

to initiating a 10-step management . system which began with the setting of^ statewide 

goals for elementary and secondary education and included the use of assessment 

, data as a way of moving from these general goals to the identification of specific 

priorities for action. Time does not permit a discussion of tllie specific operation 

of this system,^ but a few key types of decisions based on the assessment data might 

be of particular interest: . ' 

♦ Once every four years, the Board will u^e assessment data to revise 
,its state priorities. To assist with this, assessment reports are 
generated to key state advisory committees, who in turn, make recom- 
mendations about instructional and management areas needing special ^ 



^. See Appendix for ranks assigned to policy questions. 
^ See Appendix B for statement of purposes. 

^ See Appendix C for overview of Board of Education management system. 
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attention. Immediately following the adoption of the priorities, ^ 
proposed accomplishments (objectives) are developed specifying 
the type of action and results anticipated for the pext four 
J years. ^ . ^ 

• Every two years, the state priorities an^ used as the basis for ^ 
planning the Department's biennial budget and the Board's legis- 
lative program - including requests for operational funds as well 

as for grant-in-aid to targeted areas. For example, information 
collected during the pilot assessment itf reading indicated that 
some regions of the state did not have adequate access to reading 
^specialists. A grant-in-aid .request to solve this probrem was 
included in the Deparment's budget. Data also showed that students 
included in Title I of ESEA tended to score lower than other student 
groups. ^,The Board/ s 1975 Legislative Program included a recommendation 
that state monie§ be added to federal funds to provide specialized 
reading services to this population. The assessment data also influ- 
enced the Board's decision to request grant-in-aid funds for services 
to bilingual populations. 

• The Board's priorities also impact on the criteria used by the State 
Textbook Commission to review and adopt state-recommended instructional 
materials The results of the pilot assessment were used l^y the 
Commission late this fall to make decisions about the degree to 

which texts emphasizing certain reading skills should be represented 
on the adoption lists. . ■ , 

• Finally, assessment data is intended^ to impact on the^ day-to-d^y operations 
of the State Department in helping to identify particular areas to which 
services should be increased or decreased. To be certain that the reading 
results achieve this impact, the State Superintendent has directed the 
Right-to-Read staff to prepare an "action plan" specifying specific 
activities that will be undertaken during the remainder: of this year 

and during the next biennium to address some of the problem areas indi- 
cated in the results. ' This plan will affect the "mix" of state services 
such as training, on-site technical assistance, development of supplementary 
curriculum materials and sponsoring o/ special demonstration projects. 

' • I ■ • ' . 

Thus far, my comments about utilization have focused on the Board of Education, 
the Department, state advisory committees and state professional groups such as the 
TextDOok. Commission. Two other important "user" audiences are the Oregon Legislature 
and schools participating in the state assessment program. 
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It is still too early to detetmine the long-term impact that statewide 
assessment will have on decisions by these two audiences. However, preliminary 
findings are encouraging. 

• One significant piece of legislation has .been introduced in the Oregon 
Senate calling for adop.tion of a state policy on "the right- to-read" 
and requesting several million dollars during the next two years to 
address specific areas of weakness. The bill includes a request for 
reading specialists, provides for specialized services to student groups 
who did not score i/ell on the test, includes funds- for researcH and 
development to provide solutions to. questions raised by the assessment 
(i.e., why did girls score better than boys?) and calls for modifications 
in teacher training in reading. 

• Several" participating school districts have used the results to either 
question or validate results seen on their own local testing programs. 

• At least one district chose to administer the assessment instrument to ^ 
all of their fourth graders and to use the results for instructional X 
evaluation on a district-wide basis. Several additional districts are 

' expected to choose t]iis option in 1975 and the state will continue to 
encourage districts to "buy in" to expanded use of state assessment 
instruments as these meet particular -local needs. 

• Procedures and materials developed by the state assessment 'program are^ 
also being viewed by local districts as one pr6totype which may be copied 
in implementing their own local assessment design. Nejf State Minimum 
School Standards require that by the end of the 1970)^, .all Oregon 
districts will conduct student assessment of progr'am goals. 

Finally, let me brief ly review the various dissemination strategies which/have 
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been used in Oregon thus far. As I mentioned e^rjier, the planning for Oregon '.s 

assessment program has been predicated on the basis that: a) different decision- 

/ ' • - 

making audiences have, needs for different types of information, and b) this 
information must be delivered through a "medium" apprppriate to their use and 
operation. 

Basic data for this "diversjLf ied" approach was collected during July ^f 1974 
when the Northwest Regional Educational Laboratory, under contract to the Department, 
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conducted structured phone interviews with 35 individuals recommended as "repre- 
sentative" of ^key decision-uaking groups.^ These included university faculty 
Interpediate Education District and public school administrators, Depar tm.ent, of. 
Education administrators and staff, reading teachers and specialists, state advisory 
comittees, professional teacher associations, legislative staff members and staff 
working on new policies affecting reform of basjjjc school financing. Data was 

collected relating to several areas. These included: 

J' 

• Possible additional analyses 

• Degree of interest on the nine reporting variables and on statistics 
currently planned for the reports 

• Concerns specifically related to repotting assessment results 

• Questions that interviewees thought should be addrelssed in the reports 

• Reservations about the pilot assessment and suggestions for future 
assessments 

* 

To a large extent, the results from these interviews influe^nced the type .of 
dissemination strategy chosen, so let me review the type of dissemination activities 
k^yed to special audiences carried out thus far: 

General Public 

• Prior to the pubJLic release of any data, statewide panels of teachers, ^ 
parents, curriculum specialists, board members, administrators and the 
general public were pulled together to review and analyze the results 

and to make recommendations. for action to specific audiences "such as 
classroom teachers, districts, curriculum groups, the State Board and 
the Oregon Legislature. ' . 

• These recommendations, plus an overview of the assessment results, were 
presented at a statewide press conference and were included in a "popu- 
larized" report. The General Report received widespread dissemination 
throughout the state. Press releases were tailored to both state and 
regional media. According to our clipping service, the subject of reading 
received more ^ress coverage in. the 60 days following the press conference 
than at any time d^uring the last six years.' 



' Interin Technical Report: Interviews to Determine the Information Needs of Target 
Au '-^ ltjnces for the-.I973-74 Oregon Statewide Assessment Program^ report by Not thwest 
^a^^^iooal Educational Laboratory to Department of Education, August, 1974. 
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• A small brochure briefly describing the assessment program and listing 
the types of reports available to the public vas also prepared and' 

' distributed - particulat:ly in response to o6t-of -rstate reguests for. 

information. * . ' 

• A special issue ^of a Department newsletter sent to some 1,000 key 
educational decision-makers in Oregon was also devoted to the 
assessment results* Extra copies of the newsletter were made 
available to meet requests from larg« groups, such as conferences, 
where the distribution of the general report woOld have been prohibi- 
tively expensive* ^ * , • , 

State Board arid Department 

• An "executive summary" of the assessment results was prepared, for use 
by the State. Board and presented at a special^br ief ing arranged for 
this group* • * * 

• Special briefings were also, organized for Department staff, and these 
have been followed-up by more informal contact with staff specifically 
interested in the assessment results as they relate to specific populations. 

Oregon Legislature . • » 

• During fall of 1974 and winter of 1975, the assessment staff met separately 
with 15 key legislators. These meetings had two major purposes: One, to> - 
personally acquaint these individuals with the results of the pilot assess- 
ment ^nd to answer general questions they night have about th-gp program; and 
two, to seek their advice on the best method of presenting assessment data 
to the Oregon Legislature* Interestingly, freshmen legislators recommended 
that the results be disseminated through personal meetings with all key 
legislators* '<>More experienced legislators suggested that an interesting 
slide-tape presentation on the results be prepared and that briefings on 
the results be organized for key comrfiittees.^ 

• Based on these i;ecoramendations, both approaches were used. A 20 minute 
slide-tape presentation on the assessment program and the pilot testing 
results was prepared and shown at special hearings arranged for the House 
and Senate Education Coniraittees. The response was most favorable. Addi- 
tional individual meetings were also held, especially with legislators ' 
who will make the decision on the 1975-77 budget level for the- statewide 
a^sessiaent program. 



o 

Of the 15 legislators included in this study, 9 were in the House and^6 in the 
Senate. Four of the legislators were freshmen, with the remaining' eleven ranging 
from 4 to 18 years of service. A structured questionnaire was used with the legis- < 
lators giving them several alternative modes for dissemination of assessment results. 
All four freshmen legislators picked "personal meetings" as the-single most effective 
^ means of communicating assessment results. Specially-arranged committee hearings, 

a slide-tape presentation, brief written reports and personal meetings were selected. 
In rank order, by the more experienced legislators, with the most experienced members 
consistently recommending the committee hearing as the single most effective method 
ERiC of shar mg assessment results. 
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All members of the House and Senate Education conmitcees received 
^ copies of the Executive Summary and the General Report, along with 

the copy of the special newsletter. All other members of the -Legislature 
received copies of the General Report. 

<> 

Statewide Organizations 

• The Ganeral R-port and'a^special briefing tailored to their pending- 
decision on reading texts, was given to the Oregon State Textbook 
Commission. 

Copies of the newsletter were provided to the Oregon Education Association- 
for -distribution to their 400 local unit officers. A special meeting c 
was also held with the governing board of OEA to discuss the assessnent 
program -and secure input for futGre planning., tn addition, the assess- 
_ raent staff will be participating in two separate workshops to be jointly 
- sponsored by the Department and OEA later this spring to discuss statewide 
movemei^ts towards accountability. 

r 

Copies of either the general report or 'the newsletter were also nade available 
to key statewide advisory committees, such as the Intergroup Hunan Relations 
Council and the" Right-to-Read Committee, as well as the Statewide Assessment 
Advisory Committee. 

School Audiences ' ^ 

• Specialized reports were prepared for the participating schools allowing 
xor an analysis of results on both a school and individual student basis 
A user manual" was developed and distributed, providing information on 
how to interpret the results and also cautioning against misuse and niscom- 
munication of the data. Schools had the option of having, the results sert 
only to the district, to the school administrator, to classroom teachers, 
to all three or to none of the above. 

• The, General Report was distributed lo all other interested school audiences. 

• Three technical reports were also prepared - one covering general infor-^ation 
on the planning and background of the program, the second providing details ' 
on the assessment procedures and results, and the. third covering interpreta- 

. txon and recommendations. These were shared with limited educational audiences 
primarily planning and evaluation sta<=Is in the schools.'" 

The assessment staff is also currently working with the .Right- to-Read staff 
on the feasibility of issuing a specialized manual for reading teachers. This 
manual would tie assessment results to specific recommendations, for corrective 
instructional techniques and- materials. 

Speeches on the assessment program and pilot year results have also been given 
or are^ scheduled at a wide variety of educational meetings, including sessions 
with 'the Oregon School Boards Association, statewide administrator groups, 
. equal opportunity interest groups and language arts councils. . 

11 



The dissemination and utilization plans for the Oregon Statewide Assessment 
program are still evolving and will undoubtedly be changed as more experience 
and data becomes available. Throughout this process, however, the Oregon staff 
has become increasingly coramibted to the notion that the expense and effort of 
statewide assessment can only b^ justified if: 

• There is clearcut agreeme-.^. upon the purposes of the programs; 

States carefully design decision-making processes to be certain 
that the data is actually used for these purposes; 

• Adequate staff and financial resource's are provided for effective 
utilization and dissemination; 

•4 

• Dissemination strategies are carefully chosen to get the informa- 
tion in the most usable and appropriate way to differentiated 
audiences. 

' C ■ - 

Aftfer all, aren't these four eleme/vCS what "accountability" ife all about? 
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RAflKS ASSlGNp TO POLICY QUESTIONS 




1. 


lo v.hat extent have pupils attained skills,, knowledges, and attitudes related to 


i 




the six life roles identified in the OBE goals? 


2--r 


'2. 


7o vhat extent have pupils attained skills (competencies) elaborated in the Oregon 

« 

Qracuation Requirements? 


4 


3.* 


To vhat extent are pupils achieving competencies set forth in the mininun standards 
as elaborated in the High School Graduation Requirements? 


i4 


A. 


lo vhat extent are Oregon's students achieving knowledge skills as compared to 
students in the nation? 


13 


5. 


UTiat are the school district characteristics which reflate to differences in pupil 
achievement? e.g*, district s.ize, location,, wealth. 


7, 


6. 


Vihat are the school characteristics which relate to differences in pupil achieve- 
nent? e.g., school size, number of teachers, type of program. 


16 


7. 


UTiat are the community^ characteristics which relate to differences in pupil 
achievenent? e.g., urbanity, percent of welfare. 


11-12 

» 


8. 


Waat are the pupil characteristics which relate to differences in pupil achieve- 
nent? e.g-, sex^ race, nuTiber of brothers and sisters, mobility. 


15. 


9. 


Vibat are the family characteristics (SES) which relate to cafferences in pupil 
achievement? e.g., family income, parents'- education. 


6 


10. 


10 v;hat extent are achievement levels changing (increasing or decreasing) in 
instruction-related areas identified as OBE priority need's? 


10 


11. 


What additional instruction-related areas should be identified as OBE priority needs?'! 


11-12 


12-. 


Unat achievement-related district, school, community, family, and pupil characteristics 
should teacher in-service programs emphasize to increase pupil achievement? 


8 


13. 


Wliat subjject matter content should be emphasized in in-service programs to 

J. 






strengthen pupil achievement? ^ . - 


5 


4-- 

Mr. 


In vhat subject matter content is pupil achievement lowest? 


2-3 ' 


To what extent are achieveme^nt levels changing (increasing or decreasing) in the ^ 
instructional areas* receiving emphasis'? 

.J ' 


9 


16. 


In what goal areas is pupil achievement lowest? 


ERIC. . 
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OUR PROBLEM-SOLVING SYSTEr.i 



To cichievc :he sta:,..-."d: tjw. accomplish the Board pf;or;:os. gcc! bssrd piannfogi 
been intfOcLiccc! 1 1 »? c&'i^x? the Dep.artrncnrs problem solving s, sr-em. Each process of i 
system is outlifivis ^e'j, 'ojciher v/ith the procecKires for implo^eniirg the system and 
resulting produce 



PROCESS 



PF^OCEDURSS 



PRODUCTS 



1. SET (or revise) GOALS 



Identify societal nsadi for educctlonal p-o- 
g'a.Tis through advisory co.TifPittJes, pub»*c 
foru.T.s. arid legislative and Board heorlrgs. 

Trar.s'2;e identified nead> into statewide edj"c- 
tional goals; secure public validation. 



ASSESS CURRENT . .Conduct regular state-.vlde testing to meesjre 
STATUS OF OREGON'S stucJen: progress toward achieving edacatlonsJ . 
SCHOOL SYSTEM gdals. 

\ 

Collect statistical Information aboi/t conditions 
which ^affect school district ability to he^p 
students^achTeve educational g^^ls. 

Collect statistical and other Informatlori (e.g., 
fjgyres on ernployment, accidents or health fcr 
leenagers and vyoung adults] which Indlcc:r< 
ability of students to function in life roles. 



Oregon Revised 
u:£s 



Oregon State Boe? 
Educctlon Goals ; 
Elemer.tery and Sf 
dery Education 



State 
reports 



a ssessn 



Ascessinent guidd 
and models {ti 
development) 

Statistical summarl^ 

Indicators and ^ 
Y/tde Assessml 
(1973 survey I 
report)" 



3. IDENTIFY SCHOOL 
SYSTEM NEEDS; 

4. ASSIGN PRIORITIES; 

5. ESTABLISH 
OBJECTIVES FOR 
MEETING PRIORITY 
NEEDS 



Analyze assessment data to determine specific 
areas In v;hich school district progrems may ba 
Improv'ed. 

Recornmend to the State Board the needs the: 
should be given prloSi^:^ 

Identify v.-ha: changes shou?d be made (pr- 
oposed occornplishments> to effect nesdcd 
imprcvenr.en:s. 

Ryblish planning Statements which desc^-Ibe t?*2 
priority needs and proposed accompllshme'?:; « 



Encourage school districts to use 5t2:e-?e.-e! 
planning statements as gulden for develop'-g 
similar statements of commitment to achieving 
improvements In priority areo. 



Indicator data 1 
(under development 

Emerging Trends ; 
Edu cazh.'zai Plan 
(monthly newsieti 

Oregon State -Boafl 
Education Prioi 
and Proposed Ac^ 
p'ishnr.e-.ts i 

I 

Planning St at em J 
Career Educar*on\ 

\ 

Dfgnny and V/onh\ 

Emphash on Prev^n 
(readlr-gJ' j 

t.'nprovir.g Pftmar^\ 
vcazion* i 

V/izhin Reach (^ 
munlty college dj 
opnr^ent)* j 

Systems Seven, 
(school finance)* j 

Educ^itonal Pcrsdi 
OcvetoprTi^nt^ 



6. IDENTIFY ALTER- 
NATIVES 

7. SELECT PROGRAV 
PLANS 

8 ALLOCATE 
'-flESOUP{Cr.S 
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Assign responsibility for occompJishlng prio'iiy 
objectives to appropriate deportment p-og-am 
directors who: 

Id) Investigate alter native methods for accom- 
plishing objectives. 

lb) Study thif cost cffecrlvencss of' the oitemif 
tlves and select the most fcusibSe. 



PromtStr.j Pmcticei 
Oregon Educationl 
Adopter's Cwdes" j 



??:oc = ss pROCfcDuriES products 



(c) Piopo^^ bu'Jy.i 2!loco;ion> tor v.a'i t " ^. 
ma:efo!5. cq-'p'^CfU and f^icUsiiSf^. 

(d) Justify r-;qi:?*:-;d oMocations ir. bu'd^?: r<f Bi^nnra! txio'g^c 
views by t'nv S:y,-£ Board, the Gove.TC s'^d 

the Lcj'SJj'u'r 

c. V^,rV = M AfiO Assign respons'b?J::y for meetinQ pfoc'STk ob' . Curriculum" Gui:?^s Aci- 

s.^7£ rfiOG.^Afh jeciivtfS to p;o5'cra specbtist. Develop ir^di- - minis traiion r/is'j3i3 

f^i'.S vidu2^ work p^a'is. Schoof Ls.vs^ 

Use work p^ans as bcsis for monitoring 5?-,er2! Reso-jrce handbooks 

progress toward objectives and evaluatir.g ind^ 

vidu2l perfofmsnce. Position papsrs 

In-service programs 

A dm tn Istretiv^ Bates 
for Oregon Education'^ 

10 EVALUATE Identify actual p!an outcomes. " Dspartment of Educe- 

Pr?OGr*A:/» ' If on Performance indt' 

Pl,A*o A'.'D Identify discrep3nci2S between tha actL-al and cator Report 

C jTCCV.HS planned O'jrcomes. 

Oregon State Board of 

: Identify needed changes in plan, | Education Bienntst 

^ Report 



I- 



Cop i; cf p'jbH cations may be obrairred by writing to Mrs. Ellen Hassler, Docum=:n:s 
C'e.'*<. Oregon Department of Education, 942 Lancaster Dri/e NH. Sslem. OrecDn 
973 IC Pnone 378-3589. For information on other publicst.'ons and products, write to 
f-ir CcT.'nunications Section at the above address. % 
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